Interaction with Registered Displays

I magine standing in front of a real estate agent’s shop window one evening. Looking at the computer driven display in the window, you see several interesting properties. You want to download the related information and leave your details so the staff can contact you to arrange viewings. You also want to scroll through the other properties, using all the features of the large public display. Of course, you want to interact with the display directly, and seamlessly move data between it and your own smart phone—and even have your phone become part of the real estate agent’s display. Composing the two displays would circumvent the need to learn new mappings and interaction techniques and, in essence, you could directly interact with this public display through your smart phone’s screen. Indeed, by observing users performing tasks on other classes of situated displays, such as digital tabletops, researchers have noted the !uid mix of activities and rapid switching between tasks.1 Composing the displays means not having to switch between the smart phone and the public display, and it leverages our familiarity with smart phone input techniques (such as button con"gurations and stylus). In fact, the technologies necessary to realize the direct interaction in our real estate agent example are standard components of all smart phones: a rear-mounted camera and wireless connectivity. If we point the smart phone camera at a public display, we can treat the public display and the smart phone image as the same visual element, providing that we can communicate data and interaction events between the phone and the display, and quickly and reliably compute the geometric mapping between the smart phone’s image and the public display. Finding this geometric mapping is an image registration problem that we call display registration. If we can register the two displays, we can open the door to a host of potential interaction techniques, from the devices’ direct manipulation metaphors to the smart phone’s use as a physical tool or 6-degree-of-freedom (DOF) !ying mouse. This assumes a trusted and secure wireless communication between the smart phone and public display. In this respect, a related and potentially problematic issue is the usability of Bluetooth con"guration and authentication. However, recent work on spontaneous security is beginning to address these issues.2 (For other work in bridging the gap between personal devices and situated displays, see the “Related Work in Mobile Interaction” sidebar.)